COMMENT ⊗   VALID 00004 PAGES
C REC  PAGE   DESCRIPTION
C00001 00001
C00002 00002	%\read16 to\pagenumber
C00004 00003	\tracingpages=1
C00020 00004	% now we print the answers, if any
C00021 ENDMK
C⊗;
%\read16 to\pagenumber
\let\MFmanual=\! \input manmac[tex,dek]
\tenpoint
\pageno=800
%\pageno=\pagenumber

\outer\def\beginchapter#1 #2#3. #4\par{\global\exno=0
  \subsecno=0
  \def\chapno{#2#3}
  \ifodd\pageno
    \errmessage{You had too much text on that last page; I'm backing up}
    \advance\pageno by-1 \fi
  \def\\{ } % \\'s in the title will be treated as spaces
  \message{#1 #2#3:} % show the chapter title on the terminal
  \xdef\rhead{#1 #2#3: #4\unskip}
  \advance\pageno by 1 % we skip the title page for now
  \tenpoint
  \noindent\ignorespaces} % the first paragraph of a chapter is not indented
\def\rhead{Experimental Pages for The \TeX book}
\def\chapno{ X}
{\catcode`\%=12 \immediate\write\ans{% Answers for galley proofs:}}
\tracingpages=1
\beginchapter Chapter 6. How \MF\\Reads What You\\Type

So far in this book we've seen lots of things that \MF\ can do, but we haven't
discussed what \MF\ can't do. We have looked at many examples of commands that
\MF\ can understand, but we haven't dwelt on the fact that the computer will
find many phrases unintelligible. It's time now to adopt a more systematic
approach and to study the exact rules of \MF's language. Then we'll know what
makes sense to the machine, and we'll also know how to avoid ungrammatical
utterances.

A \MF\ program consists of one or more lines of text, where each line is made
up of letters, numbers, punctuations marks, and other symbols that appear on
a standard computer keyboard. A total of 95 different characters can be used,
namely a blank space plus the 94 visible symbols of standard ↑{ASCII}.
\ (Appendix~C describes the American Standard Code for Information
Interchange, popularly known as ``ASCII,'' under which code numbers 33
through~126 have been assigned to 94 specific symbols. This particular
coding scheme is not important to a \MF\ programmer; the only relevant thing
is that 94 different nonblank sysmbols can be used.)

\MF\ converts each line of text into a series of {\sl ↑{tokens}}, and it's
important to understand exactly how this conversion takes place. Tokens are
the individual lexical units that govern the computer's activities; they are
the basic building blocks from which meaningful sequences of instructions
can be constructed. We discussed tokens briefly at the end of the previous
chapter; now we shall consider them in detail. Line~9 of the file |io.mf|
in that chapter is a typical example of what the machine might encounter:
\begintt
beginchar("O",0.8em#,cap#,0); "The letter O";
\endtt
When \MF\ reads these ASCII characters it finds sixteen tokens:
\begintt
beginchar  (  "O"  ,  0.8  em  #  ,
 cap  #  ,  0  )  ;  "The letter O"  ;
\endtt
Two of these, |"O"| and |"The| |letter| |O"|, are called {\sl↑{string tokens}\/}
because they represent strings of characters. Two of them, `|0.8|' and `|0|',
are called {\sl↑{numeric tokens}\/} because they represent numbers. The
other twelve---`|beginchar|', `|(|', etc.---are called {\sl↑{symbolic
tokens}\/}; such tokens can change their meaning while a \MF\ program runs,
but string tokens and numeric tokens always have a predetermined significance.
Notice that clusters of letters like `|beginchar|' are treated as a unit;
the same holds with respect to letters mixed with ↑{underline} characters,
as in `|mode_setup|'. Indeed,
the rules we are about to study will explain that clusters of other
characters like `|0.8|' and `|:=|' are also considered to be
indecomposable tokens. \MF\ has a definite way of deciding where one
token stops and another one begins.

It's often convenient to discuss grammatical rules by formulating them in
a special notation that was introduced about 1960 by John ↑{Backus} and
Peter ↑{Naur}. Parts of speech are represented by named quantities in
↑{angle brackets}, and {\sl↑{syntax rules}\/} are used to express the ways
in which those quantities can be built~up from simpler units. For example,
here are three syntax rules that completely describe the possible forms of
numeric tokens:
\def\\#1{\thinspace{\tt#1}\thinspace}
\beginsyntax
<decimal digit>\is\\0\alt\\1\alt\\2\alt\\3\alt\\4\alt\\5\alt\\6%
  \alt\\7\alt\\8\alt\\9
<digit string>\is<decimal digit>\alt<digit string><decimal digit>
<numeric token>\is<digit string>\alt[.]<digit string>
  \alt<digit string>\\.<digit string>
\endsyntax
The first rule says that a \<decimal digit> is either `|0|' or `|1|' or
$\cdots$ or `|9|'; thus it is one of the ten numerals. The next
blah blah blah blah blah blah blah blah
blah blah blah blah blah blah blah blah
blah blah blah blah blah blah blah blah
blah blah blah blah blah blah blah blah
blah blah blah blah blah blah blah blah
blah blah blah blah blah blah blah blah
blah blah blah blah blah blah blah blah
blah blah blah blah blah blah blah blah
blah blah blah blah blah blah blah blah
blah blah blah blah blah blah blah blah
blah blah blah blah blah blah blah blah
blah blah blah blah blah blah blah blah
blah blah blah blah blah blah blah blah
blah blah blah blah blah blah blah blah
blah blah blah blah blah blah blah blah


\endchapter

A first quotation.
\author A FIRST ↑{AUTHOR},  {\sl A First Source\/} (18xx)

\bigskip

A second quotation.
\author A SECOND ↑{AUTHOR},  {\sl A Second Source\/} (19xx)

\eject
% now we print the answers, if any

% that blank line will stop an unfinished \answer
\immediate\closeout\ans
\vfill\eject
\ninepoint
\input answers
\bye